Papers with contextual representations
Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study of the effect of visual grounding on language representations has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. |
| Approach: | They compare word embeddings from vision-and-language models to text-only models . they identify meaning properties and relations that characterize words whose embeddements are most affected by visual grounding . |
| Outcome: | The proposed model differs from text-only models on semantic representations of language . the study is the first large-scale study of the effect of visual grounding on language representations . |
Detecting Gang-Involved Escalation on Social Media Using Context (D18-1)
Copied to clipboard
Serina Chang, Ruiqi Zhong, Ethan Adams, Fei-Tzin Lee, Siddharth Varia, Desmond Patton, William Frey, Chris Kedzie, Kathy McKeown
| Challenge: | In cities such as Chicago, gang-involved youth have increasingly turned to social media to post about their experiences and intents online. |
| Approach: | They propose a system that uses domain-specific resources and contextual representations of the emotional and semantic content of the user’s recent tweets and their interactions with other users to detect Aggression and Loss in social media posts. |
| Outcome: | The proposed system improves on a large unlabeled dataset and incorporates contextual representations of the emotional and semantic content of the user’s recent tweets as well as their interactions with other users. |
A Frustratingly Easy Approach for Entity and Relation Extraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on end-to-end relation extraction models combine two tasks: named entity recognition and relation extraction. |
| Approach: | They propose a pipelined approach for entity and relation extraction that uses two independent encoders to construct the relation model. |
| Outcome: | The proposed approach achieves an 8.16 speedup with a slight reduction in accuracy on standard benchmarks. |
A Non-Linear Structural Probe (2021.naacl-main)
Copied to clipboard
| Challenge: | Probing is a method of investigating the encoding of knowledge in contextual representations. |
| Approach: | They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe. |
| Outcome: | The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages. |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
NeuSpell: A Neural Spelling Correction Toolkit (2020.emnlp-demos)
Copied to clipboard
| Challenge: | a new spelling correction toolkit is available for free. |
| Approach: | They propose an open-source toolkit for spelling correction in English . they train neural models using spelling errors in context and using richer contextual representations. |
| Outcome: | The proposed spell-checker improves accuracy on synthetic examples and richer representations of the context. |
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020.acl-demos)
Copied to clipboard
| Challenge: | Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. |
| Approach: | They propose a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. |
| Outcome: | Using exBERT, we can analyze the representations and attentions of large language models and extend them to previously not analyzed models. |
Changing the Basis of Contextual Representations with Explicit Semantics (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing transformer-based contextual representations are opaque as their latent dimensions are not directly interpretable. |
| Approach: | They propose an algorithm where the output representation expresses human-interpretable information of each dimension. |
| Outcome: | The proposed transformations are able to predict supersense category of a word by looking for its transformed coordinate with the largest coefficient. |
S3 - Semantic Signal Separation (2025.acl-long)
Copied to clipboard
Márton Kardos, Jan Kostkan, Kenneth Enevoldsen, Arnault-Quentin Vermillet, Kristoffer Nielbo, Roberta Rocca
| Challenge: | Recent efforts to incorporate contextual representations into topic models have been shown to outperform classical topic models. |
| Approach: | They propose a theory-driven topic modeling approach that decomposes contextualized document embeddings into a Python package that implements S3 and all contextual baselines. |
| Outcome: | The proposed model is 4.5x faster than the BERTopic model and provides diverse and highly coherent topics with no preprocessing. |
How much do contextualized representations encode long-range context? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies of contextualized representations focus on short sequences of tens to hundreds of tokens, whereas modern language models handle hundreds of thousands of token in a single context window. |
| Approach: | They use a perturbation setup and a metric to capture contextualization of long-range patterns from the perspective of representation geometry. |
| Outcome: | The proposed model can encode long-range contexts, but it's not fully recurrent, the authors say . their results suggest improvements in existing language models . |
Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful on a wide range of NLP tasks . however, contextual representations from pre-trated models contain entangled semantic and syntactic information. |
| Approach: | They propose a semantic sentence embedding model that disentangles semantics and syntax from pre-trained models. |
| Outcome: | The proposed model outperforms state-of-the-art models on unsupervised semantic similarity tasks. |
Probing as Quantifying Inductive Bias (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextual representations have led to performance improvements on downstream tasks. |
| Approach: | They propose a Bayesian framework that quantifies the amount of inductive bias that the representations encode on a specific task. |
| Outcome: | The proposed framework alleviates many problems found in probing and can offer better inductive bias than BERT. |
Will it Unblend? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Blends, such as “innoventor”, are one particularly challenging class of OOV terms, as they are formed by fusing together two or more bases that relate to the intended meaning in unpredictable manners and degrees. |
| Approach: | They propose to use a dataset of English OOV blends to quantify the difficulty of interpreting the meanings of blends by large-scale contextual language models such as BERT. |
| Outcome: | The proposed model outperforms character-level and context-free embeddings, although their results are still far from satisfactory. |
Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work shows document-level contexts can significantly improve Named Entity Recognition models. |
| Approach: | They propose to find external contexts of a sentence by retrieving and selecting a set of semantically relevant texts through a search engine with the original sentence as the query. |
| Outcome: | The proposed approach can achieve new state-of-the-art performance on 8 NER data sets across 5 domains. |
Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models assume they encode metaphorical knowledge useful for NLP systems. |
| Approach: | They propose to probing metaphoricity information in PLMs and measure their generalization . they find that contextual representations in PMLs encode metaphorical knowledge . |
| Outcome: | The proposed model can encode metaphorical knowledge across languages and datasets . the model can be used to train and test NLP systems . |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations. |
| Approach: | They propose a representation learning approach that uses embeddings as anchors to model contextual representations. |
| Outcome: | The proposed model achieves 5x speedup and 1.2 points average improvement over MLM. |
Probing the Category of Verbal Aspect in Transformer Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a particular challenge is posed by ”alternative contexts” where either the perfective or the imperfective aspect is suitable grammatically and semantically. |
| Approach: | They investigate how pretrained language models encode the grammatical category of verbal aspect in Russian. |
| Outcome: | The proposed model has high predictive uncertainty about aspect in alternative contexts, the authors show . |
Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training . |
| Approach: | They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments. |
| Outcome: | The proposed extensions improve compositional generalization on OOD compositions. |
Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction (2021.findings-acl)
Copied to clipboard
| Challenge: | Bilingual Lexicon Induction (BLI) aims to map words in one language to their translations in another. |
| Approach: | They propose a mechanism to combine static word embeddings and contextual representations to utilize the advantages of both paradigms. |
| Outcome: | The proposed method improves performance on supervised and unsupervised BLI benchmarks on all language pairs by average improving 3.2 points over baselines. |
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension (N19-1)
Copied to clipboard
| Challenge: | Existing models for Machine Reading Comprehension (MRC) are small, compared to their size, and there are many studies on using pre-trained word embeddings and back-translation approaches to improve model generalization. |
| Approach: | They propose a multi-task learning framework to learn a machine reading comprehension model that can be applied to a wide range of MRC tasks in different domains. |
| Outcome: | The proposed model can be applied to a wide range of MRC tasks in different domains. |
Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)
Copied to clipboard
| Challenge: | a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers . |
| Approach: | They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span . |
| Outcome: | The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings. |
Comparing Static and Contextual Distributional Semantic Models on Intrinsic Tasks: An Evaluation on Mandarin Chinese Datasets (2024.lrec-main)
Copied to clipboard
| Challenge: | Distributional Semantics has undergone significant changes with the introduction of contextualized distributional models. |
| Approach: | They compare static and contextual distributional models for Mandarin Chinese . they find that static models are stronger for some of the classical tasks . |
| Outcome: | The proposed models perform better on some of the classical tasks that consider word meaning independent of context, while contextualized models excel in identifying semantic relations between word pairs and categorization of words into abstract semantic classes. |
Deep Inside-outside Recursive Autoencoder with All-span Objective (2020.coling-main)
Copied to clipboard
| Challenge: | Existing neural approaches for constituency parsing are limited for low-resource languages and domains. |
| Approach: | They extend the training objective of DIORA by making use of all spans instead of only leaf-level spans. |
| Outcome: | The proposed model improves on two languages and provides better parsing accuracy than the original model. |
Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that contextualized models generate dynamic embeddings for words in context, but static embedds are often overlooked in this trend towards contextualized modeling. |
| Approach: | They propose a method that learns a transformation through static anchors and requires only another pre-trained model. |
| Outcome: | The proposed method improves a range of benchmark tasks that test contextual variations of meaning across different usages of a word and across different words as they are used in context. |
Don’t Just Clean It, Proxy Clean It: Mitigating Bias by Proxy in Pre-Trained Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Transformer-based pre-trained models can encode societal biases in their contextual representations and in downstream predictions when fine-tuned on task-specific data. |
| Approach: | They propose an approach that selectively eliminates stereotypical associations at fine-tuning, so that the model doesn't learn to excessively rely on those signals. |
| Outcome: | The proposed approach reduces biases from identity words and frequently co-occurring proxies by > 60% in toxicity classification, and also extends to multiple identities. |
Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on whether multilingual embeddings can be aligned in a shared space across languages are lacking. |
| Approach: | They propose to learn a projection based on monolingual annotated datasets and evaluate syntactic and lexical information encoded in a shared cross-lingual embedding space. |
| Outcome: | The proposed model can be used to learn representations for languages with low resources. |
Active Learning for Rumor Identification on Social Media (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for rumor tracking depend on a significant amount of labeled data. |
| Approach: | They propose an Active-Transfer Learning strategy to identify rumors with limited amount of annotated data. |
| Outcome: | The proposed approach achieves faster convergence in terms of the F-score while requiring fewer annotated samples (42% of the whole dataset for the best model). |
Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to predict emotion label for a given utterance lack modeling of diverse dependency ranges and inconsistent treatment of contribution for various modalities. |
| Approach: | They propose a multimodal emotion recognition in conversation task that uses context and multiple modalities to predict emotion label for a given utterance. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on three multimodal datasets. |
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones. |
| Approach: | They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets. |
| Outcome: | The proposed model can be trained on multiple RC datasets and improve performance on five RC data. |
A Contextual Hierarchical Attention Network with Adaptive Objective for Dialogue State Tracking (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for dialogue state tracking ignore the slot imbalance problem and treat all slots indiscriminately, which limits the learning of hard slots. |
| Approach: | They propose to employ a contextual hierarchical attention network to enhance the DST by learning contextual representations. |
| Outcome: | The proposed approach achieves 52.68% and 58.55% joint accuracy on multiWOZ 2.0 and MultiWOZ 2.1 datasets and significantly improves performance (+1.24% and +5.98%) |
Pre-Training BERT on Domain Resources for Short Answer Grading (D19-1)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations have achieved state-of-the-art results on multiple downstream NLP tasks by fine-tuning with task-specific data. |
| Approach: | They propose to augment domain-specific data by using labeled short answering grading data for further enhancement of the pre-trained language model. |
| Outcome: | The proposed model can be enhanced by augmenting data from domain-specific resources like textbooks and labeled short answering grading data. |
Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work on predicting backchannel timing has focused on lexical form and prosody, but the relationship between lexico-prosodic form and meaning remains underexplored. |
| Approach: | They propose a framework for fine-tuning large language models on dialogue transcripts to derive rich contextual representations; and a joint embedding space for dialogue contexts and backchannel realizations. |
| Outcome: | The proposed framework improves context-backchannel retrieval and human perception is more sensitive to extended conversational context and embeddings align more closely with human judgments than raw WavLM features. |
MelTrim: Coarse-to-Fine Data Pruning for Speech Classification (2026.findings-acl)
Copied to clipboard
Shaobo Wang, Tianle Niu, Xuan Ouyang, Xintong Li, Zhengkun Ge, Yue Min, Xiaoqian Liu, Hankun Wang, Linfeng Zhang
| Challenge: | Unlike image or text classification, speech classification tasks are particularly challenging due to the difficulty in capturing the acoustic, semantic, and contextual representations. |
| Approach: | They propose a dataset pruning method that coarsely filters redundant samples using DBSCAN clustering on Mel-Frequency Cepstral Coefficients (MFCC) features. |
| Outcome: | The proposed method achieves 49.5% improvement in WA on the MEAD dataset and 41.9% reduction in EER on speaker identification tasks. |
Enhancing the Context Representation in Similarity-based Word Sense Disambiguation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing similarity-based systems focus on learning sense embeddings using only the sentence where the word appears, neglecting its global context. |
| Approach: | They propose a contextoriented embedding technique that takes better advantage of both word-level and sense-level global context of an ambiguous word for disambiguation. |
| Outcome: | The proposed method improves on all-words WSD benchmarks in knowledge-based category by large margins. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
The Architectural Bottleneck Principle (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examined how much information a model's representations contain . a new approach to probing is to look exactly like the component . |
| Approach: | They propose a new probing principle that aims to estimate how much information a model could extract from its representations. |
| Outcome: | The proposed probes extract syntactic information from the representations of a neural network . the proposed probe is based on the architectural bottleneck principle . |
KG-CQR: Leveraging Structured Relation Representations in Knowledge Graphs for Contextual Query Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that address corpus-level context loss focus on query enrichment through structured relation representations. |
| Approach: | They propose a framework for Contextual Query Retrieval that enriches queries with contextual representations derived from a corpus-centric KG. |
| Outcome: | The proposed framework outperforms strong baselines on RAGBench and MultiHop-RAG datasets in terms of retrieval effectiveness. |
TransLLM: A Unified Multi-Task Large Language Model for Urban Transportation via Learnable Prompting (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models lack generalization capabilities and lack structured spatiotemporal data. |
| Approach: | They propose a unified multi-task framework that synergizes spatiotemporal encoding with LLM reasoning through learnable prompt composition. |
| Outcome: | The proposed framework outperforms baseline models on seven datasets and three tasks on supervised and zero-shot settings with excellent generalization and robustness. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |